1. Field of the Invention
The present invention relates generally to digital image tone mapping algorithms, and specifically to digital image tone mapping algorithms based on perceptual preference guidelines.
2. Description of Related Art
Digital image tone mapping refers to the process of mapping captured scene luminance levels to the luminance or density levels of an output device (display or printer). Digital image tone mapping is necessary due to the fact that scene luminance ranges very seldom match the luminance range of the output device. Digital image tone mapping algorithms can be implemented within any digital device to produce visually pleasing output images. For example, such devices can include digital cameras, scanners, digital camcorders, printers capable of printing digital images and digital televisions. In addition, digital tone mapping algorithms are frequently used in image enhancement software applications.
Since the human eye adapts to different light levels in different ways, perceptual factors must be taken into account when attempting to generate a rendered image that looks “right” with respect to the original scene. Depending on the application, image tone mapping typically has one of three goals: (1) appearance match; (2) subjective preference; or (3) information preservation. The goal of appearance matching strives to make the rendered image as perceptually similar as possible to the original scene. This is usually an implicit goal in consumer imaging and image synthesis applications. Taking into account subjective preferences allows the image to look as pleasing as possible to the viewer. This is usually desirable in consumer imaging and commercial photography. If the goal is information preservation, the algorithm seeks to preserve or enhance the details at all regions and all luminance levels of an image. This is most often requested in medical imaging, satellite imaging, and archiving.
Many existing tone mapping algorithms focus on achieving an appearance match between the original image and the rendered image. There are at least two perceptual factors typically considered in such algorithms: (1) global luminance adaptation; and (2) local luminance adaptation. The overall global luminance level of the scene influences the adaptation state of the eye. Two aspects of such global luminance adaptation have significant impact on tone mapping, brightness and spatial contrast.
First, the brightness function is different at different scene luminance levels. Perceived brightness, which corresponds to the viewer's perceived strength of light, is roughly a power function of physical luminance (Steven's law). The exponent for such a power function is larger when the overall luminance level is higher. When rendering an image on a lower luminance device, the exponent must be adjusted to accommodate such differences. In addition to the brightness function change, the spatial contrast sensitivity of the eye also changes as it adapts to different ambient luminance levels. When the ambience is bright, the eye perceives the high spatial frequency components (details) of an image better than when the ambience is dark, i.e., the visual acuity of the eye improves with better ambient lighting. Also, the contrast threshold, i.e., the minimum contrast needed to detect components of the image, decreases with increased luminance level. To render a bright image onto a lower luminance device, the luminance contrast of the details in the image can be enhanced to account for these effects.
Two different tone mapping algorithms developed by Jack Holm and Tumblin & Rushmeier, respectively, account for the brightness function change by adjusting the curvature of the tone curves based on the absolute luminance level of the scene. Each of these tone mapping methods is described separately in Holm, J., “Photographic Tone and Colour Reproduction Goals,” CIE Expert Symposium on Colour Standards for Image Technology, pp. 51–56 (1996); and Tumblin, J. and Rushmeier, H., “Tone Reproduction for Realistic Images,” IEEE Computer Graphics and Applications, 13(6):42–48 (1993), both of which are hereby incorporated by reference. These algorithms have the benefit of creating the proper overall sensation of brightness or darkness corresponding to the original image, which is desirable in high-end digital imaging. However, both require accurate information about the absolute luminance level of the original image. In a digital camera, it is possible to estimate absolute luminance levels of image pixels from the raw pixel values and the camera's capture settings such as aperture, exposure time, lens properties, etc. However, in low cost cameras, such calculations are often not available due to the added complexity and cost. Therefore, a global luminance adaptation solution that focuses on preference is more practical.
In tone mapping algorithms based on preference, the goal is to achieve a certain set of image properties liked by viewers. The widely used histogram equalization method can be categorized as such an algorithm. The histogram method is based on the observation that most “good” images have a luminance histogram that fully occupies the output dynamic range. The algorithm adjusts image gray levels to move the histogram shape toward a flat, Gaussian, or some other predetermined shape. Of course, how well such a method works depends on whether the assumption is true that every “good” image has the same histogram. The method does well on images that have a symmetric and well-distributed histogram, but makes images look unnatural when there are large areas of dark or light background in the image, which bias the histogram toward one side.
A modified histogram equalization method developed by Larson, et al. is more robust than traditional histogram equalization methods. Larson's method limits the amount of gray level adjustments allowed in the tone mapping. The amount of gray level adjustments are limited based on luminance contrast sensitivity measurements. In addition, one variation of this method also accounts for the change in visual acuity under different illumination levels. Reference is made to Larson, G. W., Rushmeier, H. and Piatko, C., “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes,” IEEE Transactions on Visualization and Computer Graphics, 3(4):291–306 (1997), which is incorporated by reference. However, the computation is iterative, and thus the implementation is costly and slow. In addition, the modified histogram equalization method also requires accurate absolute luminance level information. Thus, although this modified histogram equalization method creates a more accurate appearance match, it does so at the cost of higher computational complexity.
The local luminance adaptation perceptual factor considers the fact that the eye looks at an image by scanning around. The eye can rapidly adapt to the luminance level of small regions in the original scene to enable regions in the shadows and in the highlights to be clearly visible to the eye. In the rendered image, both the dynamic range and the adaptation environment are different. Therefore, to fully imitate the eye's adaptation process, the luminance levels of an image are adjusted according to its local luminance levels.
Various local tone mapping algorithms, such as Tumblin's detail-preserving contrast reduction method (Tumblin, J. and Turk, G., “LCIS: A Boundary Hierarchy for Detail-Preserving Contrast Reduction,” Computer Graphics Proceedings, SIGGRAPH 99, pp. 83–90, Los Angeles, Calif., USA (1999), which is incorporated by reference), and various algorithms based on the retinex theory have attempted to imitate the local luminance adaptation process. Reference is made to Jobson, D., Rahman, Z. and Woodell, G., “A Multiscale Retinex for Bridging the Gap Between Color Images and Human Observation of Scenes,” IEEE Transactions on Image Processing, 6(7):965–976 (1997); and Rahman, Z., Jobson, D. and Woodell, G., “Multi-Scale Retinex for Color Image Enhancement, Proceedings,” International Conference on Image Processing, volume 3, pp. 1003–1006, Lausanne, Switzerland (1996) for a discussion of the retinex theory, both of which are incorporated by reference. Although these algorithms do preserve the local contrast of images, they are iterative methods that also involve the decomposition of different spatial resolution components of an image, which is computationally costly.